1. Technical Field
This disclosure relates to digital image processing and more particularly to a method, implementable in a relative device, for calculating global motion vectors of images of a video sequence, and relates to a method of removing motion effects from images of a video sequence, and generating corresponding motion compensated images.
2. Description of the Related Art
Video stabilization serves to compensate for unintentional motion in a video sequence, such as motion caused by user hand shaking that causes unpleasant video sequences for consumers. Various image stabilization techniques (feature based, block matching based, optical flow based, etc.) have been developed. See F. Vella, A. Castorina, M. Mancuso, G. Messina, “Digital Image Stabilization By Adaptive Block Motion Vectors Filtering”, IEEE Transactions On Consumer Electronics, vol. 48, no. 3, August 2002; A. Engelsberg, G. Schmidt, “A comparative review of digital image stabilising algorithms for mobile video communications”, IEEE Transactions on Consumer Electronics, Vol. 45, No. 3, August 1999; J. K. Paik, Y. C. Park, D. W. Kim, “An adaptative motion decision system for digital image stabilizer based on edge pattern matching”, IEEE ICCE Consumer Electronics, Digest of Technical Papers, June 1992; J. K. Paik, Y. C. Park, S. W. Park, “An edge detection approach to digital image stabilization based on tri-state adaptive linear neurons”, IEEE Transactions on Consumer Electronics, Vol. 37, No. 3, August 1991. Most of these techniques are computationally burdensome and thus unsuited for real time applications.
A simple and fast processing technique is based on motion estimation through horizontal and vertical characteristics curves. See M. Kim, E. Kim, D. Shim, S. Jang, G. Kim, and W. Kim, “An Efficient Global Motion Characterization Method for Image Processing Applications,” IEEE Trans. On Consumer Electronics, pp. 1010-1018, Nov. 1997; Y. Koo and W. Kim, “An Image Resolution Enhancing Technique Using Adaptive Sub-Pixel Interpolation For Digital Still Camera System”, IEEE Transactions On Consumer Electronics, Vol. 45, No. 1., February 2005. This is a good choice in particular under the assumption that misalignments are of limited severity and are mostly due to vertical and horizontal shifts.
These methods based on characteristics curves work as follows. For simplicity, let's assume that two gray-scale frames have been captured with two successive captures. The characteristics curves along the horizontal and vertical dimensions are respectively defined as:
                                                        C              h                        ⁡                          (              j              )                                =                                    1              N                        ⁢                                          ∑                i                N                            ⁢                              p                ij                                                    ⁢                                  ⁢                                            C              v                        ⁡                          (              i              )                                =                                    1              M                        ⁢                                          ∑                j                M                            ⁢                              p                ij                                                                        (        1        )            wherein M and N are the horizontal and vertical dimensions and pij is the pixel value in position (i,j).
The meaning of the curves can be easily understood by referring to the drawing of FIG. 1 that illustrates two exemplary characteristics curves. FIG. 2 and FIG. 3 show Ch and Cv curves for two successive frames. A shift of the curves along the x-axis represents an equivalent shift of the frames in the horizontal dimension (similar approach for y-axis and vertical shift). In FIGS. 2 and 3 the horizontal shift is particularly evident. The curves appear to be shifted also along the y-axis, maybe due to slightly changing lighting conditions. Hence, to evaluate the motion occurred between consecutive frames, the shift along the x-axis of both Ch and Cv curves may be obtained by the following formula:
                                                                        P                h                            ⁡                              (                s                )                                      =                                          1                                  M                  -                                                          s                                                                                  ⁢                                                ∑                                      j                    =                                          max                      ⁡                                              (                                                  l                          ,                                                      -                            s                                                                          )                                                                                                  min                    ⁡                                          (                                                                        M                          -                          s                                                ,                        M                                            )                                                                      ⁢                                                                                                                      C                        h                                                  F                          1                                                                    ⁡                                              (                        j                        )                                                              -                                                                  C                        h                                                  F                          2                                                                    ⁡                                              (                                                  j                          +                          s                                                )                                                                                                                                            ⁢                                          ⁢                      off            h                    =                      {                                                            s                  ′                                ⁢                                  :                                ⁢                                                                  ⁢                                                      P                    h                                    ⁡                                      (                                          s                      ′                                        )                                                              =                              min                ⁢                                                                  ⁢                                                      P                    h                                    ⁡                                      (                    s                    )                                                                        }                          ⁢                                  ⁢                                            P              v                        ⁡                          (              s              )                                =                                    1                              N                -                                                    s                                                                        ⁢                                          ∑                                  i                  =                                      max                    ⁡                                          (                                              l                        ,                                                  -                          s                                                                    )                                                                                        min                  ⁡                                      (                                                                  N                        -                        s                                            ,                      N                                        )                                                              ⁢                                                                                                          C                      h                                              F                        l                                                              ⁡                                          (                      i                      )                                                        -                                                            C                      h                                              F                        2                                                              ⁡                                          (                                              i                        +                        s                                            )                                                                                                                          ⁢                                  ⁢                              off            v                    =                      {                                                            s                  ′                                ⁢                                  :                                ⁢                                                                  ⁢                                                      P                    v                                    ⁡                                      (                                          s                      ′                                        )                                                              =                              min                ⁢                                                                  ⁢                                                      P                    v                                    ⁡                                      (                    s                    )                                                                        }                                              (        2        )            
Once horizontal offh and vertical offv displacements (offsets) between two curves of successive images are computed, by calculating the relative global motion vector (rgmv) it is possible to determine the absolute global motion vector (agmv), that is used for aligning frames belonging to the sequence:agmv=off+rgmv=(offh+rgmvx,offv+rgmvy)  (3)
In practice, when absolute global motion vectors are determined, it is possible to stabilize the video sequence by aligning all frames to a reference frame. This may be done for eliminating only unintentional motion or even for eliminating also intentional motion of the camera.
A drawback of this method consists in that it is relatively sensitive to:                illumination changes;        motion blur;        new details entering into the scene;        moving objects in the scene.        